Prioritizing memory pages to copy for memory migration

ABSTRACT

A method for transferring pages from a memory to a page repository identifies an affinity domain that associates processors and memory pages. The method further includes determining that a processor in an affinity domain is suspended and selecting pages included in that affinity domain to transfer prior to transferring pages in an affinity domain in which no processors are suspended.

BACKGROUND

The present disclosure relates to transferring contents of memory pagesin a computer, and more specifically, to transferring the contents to apage repository. More specifically, the disclosure relates todetermining a priority order of pages to transfer from among pagesselected for transfer.

SUMMARY

Aspects of the present disclosure provide a method for transferringpages from a memory. The method includes identifying two affinitydomains, in which an affinity domain associates particular processorswith a set of affinity pages within a computer. None of the processorsin a first affinity domain are in a suspended state. A second affinitydomain includes a processor in the suspended state. Features of themethod further provide for identifying two pages in the memory, bothpages to be transferred to a page repository. A first page is in thefirst affinity domain and a second page is in the second affinitydomain. The second page is also associated with a program that is notexecuting.

According to features of the disclosure, the method may prepare transferof the second page to the page repository to occur prior to transferringthe first page to the page repository. Preparing the second page totransfer prior to the first page may be based on the second page beingin the second affinity domain and the first page in the first affinitydomain, and the second page being associated with the program that isnot executing.

According to other features of the disclosure, the method identifies athird page, which is also included in the second affinity domain. Thethird page is to be transferred to the page repository. The third pageis associated with a second program, the second program had lastaccessed the third page using a processor that is presently in thesuspended state, and the suspended processor is included in the secondaffinity domain. The program is not executing on any processor in thesecond affinity domain. The method may prepare transfer of the thirdpage to the page repository to occur prior to transferring the firstpage to the page repository. Preparing the third page to transfer priorto the first page may be based on the third page being in the secondaffinity domain and the first page in the first affinity domain, thethird page accessed last by the second program, and the second programnot executing on a processor in the second affinity domain.

In some embodiments of the disclosure a computer program product mayimplement the method in a system according to the structure of thedisclosure. According to various embodiments of the disclosure a systemmay be configured to perform the method.

The above summary is not intended to describe each illustratedembodiment or every implementation of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings included in the present application are incorporated into,and form part of, the specification. The drawings illustrate embodimentsof the present disclosure and, along with the description, serve toexplain the principles of the disclosure.

FIG. 1 is a block diagram illustrating a computer including a pagerepository, according to aspects of the disclosure.

FIG. 2 is a block diagram illustrating a computer including prioritymemory pages to transfer to a page repository, according to aspects ofthe disclosure.

FIG. 3 is a block diagram illustrating a computer including memory pagesto transfer to two page repositories, according to aspects of thedisclosure.

FIG. 4 illustrates conceptual structures particular to selecting memorypages to transfer, according to aspects of the disclosure.

FIG. 5 is a flowchart illustrating an example method to prepare totransfer memory pages in a priority list prior to transferring memorypages in a transfer list, according to aspects of the disclosure.

FIG. 6 is a flowchart illustrating an example method to determine a setof suspended CPUs, according to aspects of the disclosure.

FIG. 7 is a flowchart illustrating an example method to identify memorypages to include in a priority set of memory pages to transfer to a pagerepository, according to aspects of the disclosure.

FIG. 8 is a flowchart illustrating an example method to identify memorypages to include in a priority set among memory pages to transfer to twopage repositories, according to aspects of the disclosure.

FIG. 9 is a block diagram illustrating a computer program product thatmay implement features of the disclosure.

While the disclosure is amenable to various modifications andalternative forms, specifics thereof have been shown by way of examplein the drawings and will be described in detail. It should beunderstood, however, that the intention is not to limit the disclosureto the particular embodiments described. Rather, the intention is tocover all modifications, equivalents, and alternatives falling withinthe spirit and scope of the disclosure.

DETAILED DESCRIPTION

Aspects of the present disclosure relate to transferring pages from amemory to a first page repository. More particular aspects relate toidentifying priority pages within a memory and preparing transfer of thepriority pages to the first page repository to occur prior totransferring other memory pages identified to transfer. Other aspectsrelate to identifying a first set of pages within a memory to transferto a first page repository and identifying a second set of pages withina memory to a second page repository and preparing transfer of pages inthe second set of pages to occur prior to transferring pages to thefirst page repository. More particular aspects relate to determiningthat transferring pages to the second page repository may avoidtransferring pages within the first set of pages.

Computers may feature transferring the contents of pages in a memory toa “page repository”. Computers that feature such transfer capabilitiesinclude, for example, desktop, laptop, or server computers; mobiledevices such as tablets or phones; network gateways or routers; IOadapters, including virtualized IO adapters; or other devices thatinclude a memory or storage medium. Such computers also may feature oneor more operating systems, and an operating system may transfer memorypages to a page repository. Computers may feature a hypervisor, whichmay be a program that manages the operation of a plurality of programsor operating systems, such as in a partitioned computer system. Ahypervisor may transfer pages of a memory to a page repository.

For purposes of understanding the disclosure, a “memory” refers to anystorage medium of a computer, such as a memory cache, a main memory, aflash memory, or a removable memory (e.g., CD, DVD, or flash drive). Avariety of devices or components of a computer, or computing system, mayoperate as a page repository, such as pages in the same or anothermemory, a storage medium (e.g., CD or DVD, flash memory, etc.), an IOadapter, a device included in or coupled to an IO adapter, a virtualizedform of an IO adapter or device, a network attached storage (NAS)device, an operating system (OS), or another computer. A program (e.g.,an application or a workload partition in an operating system), anoperating system (e.g., a virtual machine operating in a computer), or ahypervisor may operate as a page repository. A computer may accomplishtransferring the pages to a page repository in a variety of means, suchas by hardware devices (e.g., network interfaces, IO adapters, and DMAdevices), firmware programs (e.g., BIOS, UEFI, or a hypervisor),software programs (e.g., an operating system or hypervisor), anothercomputer, or a variety of combinations of these.

Within the scope of the disclosure, a program may be any unit ofcomputer instructions that accomplishes a particular function. Forexample, a program may be, or may be a component of, an application, anoperating system, a hypervisor, a function of a computer embodied infirmware; or applications, widgets, or other forms of programs operatingon a mobile device. A program may divide its execution into one or moresimultaneously multi-threaded (SMT) “program threads”, and a programthread may execute on different processors of a computer than otherprogram threads comprising the same program. References within thedisclosure to a “program” refer implicitly to any one or more of theprogram threads that may comprise a program, and generally to anyembodiment of a program capable of executing in a computer.

A computer may transfer memory pages to a page repository for a varietyof purposes. For example, a virtual memory function of an OS maytransfer (“page-out”) pages in a main memory to a paging device (e.g., adisk drive or other storage medium) to make the pages in the main memoryavailable to a second program. An OS may transfer pages of a memory to apaging device to save (or, “checkpoint”) the memory state of a program.A computer may feature a “workload partition” encapsulating theoperation of one or more application programs within an OS. The computermay also feature transferring a workload executing within a first OS toexecute within a second OS. Transferring the workload between OSes mayinclude memory migration, transferring the contents of memory pagesallocated to the workload in the first OS to other memory pages that areallocated to the workload in the second OS.

Similarly, a computer may feature operating system partitions (e.g.,virtual machines) encapsulating an OS, and OSes within the partitionsmay share the resources of the computer. A hypervisor may be included ina computer to manage the sharing of resources among the OSes, or tomanage the execution of the OSes. A hypervisor may transfer pages to apaging device (or, to another OS) to save (or, “checkpoint”) the memorystate of an operating system.

In another example, a computer may feature memory sharing, in which thecomputer may share memory pages among a plurality of OS partitionsoperating in the computer. In a memory sharing operation a computer, ora hypervisor operating in the computer, may transfer memory pages in useby a first OS to a paging device (or, to another OS) to make the memorypages available to a second OS. A computer may also feature partitionmigration—transferring execution of an OS—from a first computer to asecond computer. A hypervisor may perform the partition migration, ormay perform the partition migration in combination with another OS, suchas a virtual IO server OS. A partition migration may include memorymigration, transferring memory pages allocated to an OS in the firstcomputer to other pages in the memory allocated to the OS in the secondcomputer.

A computer may transfer memory pages to a page repository while an OS orprogram is continuing to execute (while the OS or program is “live”),such as in a live migration (LM) operation or in an “active” memorysharing (AMS) operation. A computer may transfer pages in use by an OSor program such that the OS or program is unaware that the pages areundergoing the transfer. A live (e.g., executing) program may modify apage after a computer has initially transferred that page to a pagerepository, and this may require that the computer transfer that page asecond time prior to releasing the page for other uses. Transferringpages more than once may degrade the performance of the program or thecomputer overall. Transferring pages less likely to be modified by anexecuting program may avoid transferring a particular page more thanonce, and thereby avoid possible related performance degradation.

Selecting pages less likely to be modified may utilize statisticalmetrics relating to access or modifications of a memory page, such asleast recently used (LRU) or least frequently used (LFU). Such metricsmay be used additionally to determine an order in which to transfer thepages. However, such metrics at times in the overall operation of acomputer, or with regard to certain programs, may not be preciseindicators of likelihood that a page will be modified. Other metrics,such as which program last accessed a page and from what processor thataccess occurred, may improve selecting pages less likely to be modified.

A processor may have a suspended state, which may include any state of aprocessor such that it is not executing instructions to perform usefulwork, such as executing “idle” or “no-op” loops. A processor may have a“folded” type of a suspended state in which a processor is placed in alow power state to save energy. For purposes of the disclosure,references to “suspended” state of a processor include a folded type ofprocessor state. A program, an operating system, or a hypervisor mayplace a processor in a suspended state. A page in a memory that was lastaccessed by processor in a suspended state may be less likely to bemodified than pages last accessed or in use by active or runningprocessors. Similarly, a program may be in an idle state, and pages lastaccessed by an idle program may be less likely to be modified than pageslast accessed by an active program.

A computer may include or be coupled to a first and a second pagerepository and may transfer memory pages concurrently to bothrepositories. For example, shared pages may be transferred to a firstpage repository, such as in in an AMS operation. Some pages, which mayinclude shared pages, may be transferred to a second page repository,such as a second computer in an LM operation. If the computer initiallytransfers pages to the first page repository (e.g., to satisfy a needfor pages in a memory sharing operation), the computer may have totransfer the pages back to the memory (“page-in”) in order to thentransfer those pages to the second page repository (e.g., a secondcomputer in an LM operation). Paging-out and paging-in pages from thefirst repository, in order to transfer them to the second repository,can be inefficient and may degrade program or overall computerperformance. Additionally, transferring the pages to the secondrepository, before transferring those pages to the first repository, mayavoid transferring the pages to the first repository at all. Forexample, pages to be migrated in an LM operation may be transferred fromthe memory and may make memory pages available for memory sharing, andmay avoid transferring pages from the memory specifically for sharing.

Accordingly, aspects of the disclosure relate to structures and methodsfor identifying pages to transfer first (priority pages) to a pagerepository, in various memory transfer operations of a computer orcomputing system.

FIG. 1 illustrates a computer 100, which may include one or moreprocessors (hereinafter, CPUs), such as CPUs 110 a, 110 b, 110 c and 110d. CPUs may be a singular CPU (e.g., a single core CPU), may be a threadof a simultaneously multi-threaded (SMT) CPU, or may be a virtual CPU,such as a fractional allocation of a physical CPU or SMT CPU. Thecomputer 100 may include a memory 130, and the memory 130 may beorganized as pages, such as pages 150 a through 150 f and 140 a and 140b.

A computer may feature affinity domains, such as affinity domains 170and 180, in which particular CPUs and particular pages of a memory areincluded in one affinity domain that may exclude other processors andother pages of memory. A property of an interconnection structure orother relationship between certain CPUs and certain memory pages, notshared by all CPUs and memory pages, may determine an affinity domain.For example, CPUs 110 a, 110 b, and 110 c may have a low latencyinterconnection to particular memory pages, such as pages 150 a through150 f, while CPU 110 d may have a higher latency interconnection tothose pages. Similarly CPU 110 d may have a low latency interconnectionto memory pages 140 a and 140 b, while CPUs 110 a, 110 b, and 110 c mayhave a high latency interconnection to pages 140 a and 140 b.Correspondingly, CPUs 110 a, 110 b, and 110 c may be included in anaffinity domain 170 with pages 150 a through 150 f, while CPU 110 d maybe included in an affinity domain 180 with pages 140 a and 140 b.Affinity domain 170 may exclude CPU 110 d, and affinity domain 180 mayexclude CPUs 110 a, 110 b, and 110 c.

It will be apparent to one of ordinary skill in the art that variousembodiments may determine an affinity domain based on other physical orlogical organizations of CPUs and memory suitable to the operation of acomputer or programs operating within that computer.

A computer 100 may have one or more OSes, such as OS 102 a and 102 b,operating within it, and an OS may have an identifier that distinguishesthat OS from other OSes. An OS may utilize particular CPUs andparticular memory pages included in an affinity domain. For example, OS102 a may utilize CPU 110 a and pages 150 a and 150 b included inaffinity domain 170. OS 102 b may utilize CPU 110 b and 110 c and pages150 c through 150 f, also included in affinity domain 170. OS 102 b mayalso utilize CPU 110 d and pages 140 a and 140 b included in affinitydomain 180. The pages utilized by an OS may form a memory image, such asmemory image 130 a including the pages utilized by OS 102 a, or memoryimage 130 b including the pages utilized by OS 102 b.

An OS may include various programs operating within it, such as program104 a operating in OS 102 a and program 104 b operating in OS 102 b. Aprogram may have an identifier uniquely distinguishing it from otherprograms within an OS. A program may have an idle state, which mayrepresent that a program is suspended from further execution or hasterminated. An idle state may also represent that a program is awaitingcompletion of some event, such as release of an SMT thread lock orcompletion of an IO operation. The pages utilized by a program, or by aprogram thread, may form a memory image, such as memory image 130 aincluding the pages utilized by program 104 a, and memory image 130 bincluding the pages utilized by program 104 b.

The computer 100 includes a hypervisor 120, and the hypervisor 120 maymanage the operations of the various CPU and memory resources of thecomputer 100. A hypervisor 120 may manage allocation of CPUs or memorypages to the various OSes, such as OS 102 a and 102 b, or programs, suchas programs 104 a or 104 b, operating in the computer 100. For example,the hypervisor 120 may allocate CPUs 110 b, 110 c, and 110 d to OS 102b. OS 102 b may allocate CPUs 110 b, 110 c, and 110 d to program 104 b.The hypervisor 120 includes a hypervisor interface 122, and the OSes orprograms and the hypervisor 120 may communicate utilizing the hypervisorinterface 122. In other embodiments, the computer 100, or anothercomponent thereof, or an OS may allocate a CPU or memory to a programfor program execution.

A program may utilize pages in the affinity domain of CPUs allocated tothe program, and may exclude pages in other affinity domains from use bythat program. For example, a program may be allocated a particularprocessor and may determine a scheduler resource allocation domain(SRAD) that includes that processor and certain memory pages that arewithin an affinity domain having that processor, such as SRAD 175 inaffinity domain 170. When executing on a processor in a particular SRAD,a program may limit memory pages it uses to only pages within that SRAD.For example, program 104 b may execute on CPU 110 b and program 104 bmay determine to use only pages 150 c and 150 d within SRAD 175 whenexecuting on processor 110 b. Program 104 b may further determine not touse pages in other affinity domains, such as pages 140 a or 140 b inaffinity domain 180, while executing on processor 110 b.

A CPU may have a suspended state, which may be any state in which theCPU is not executing instructions, or is executing instructions that maynot perform useful work but may maintain an active execution state ofthe CPU (e.g., looping while awaiting dispatch of a program). A CPU,such as 110 a, 110 b, 110 c, or 110 d, may have a “folded” kind ofsuspended state in which the CPU is in a low power state not executinginstructions. A computer, or a program may place a CPU into a suspendedstate at times when that CPU may not be actively utilized (e.g., isidle). For example, OS 102 b or program 104 b may suspend CPU 110 bwhile executing programs on another CPU, such as CPU 110 c, and CPU 110b may be idle. A hypervisor 120 may suspend CPU 110 b, or a virtual CPUexecuting within CPU 110 b, when the CPU 110 b is idle. A computer 100may suspend a CPU under conditions in which execution can beconsolidated onto fewer CPUs and the unused CPU may be suspended. Forexample, CPUs 110 b and 110 c may have sufficient capacity to executeprogram 104 b and OS 102 b may dispatch program 104 b to execute on CPUs110 b and 110 c and may place CPU 110 d in a suspended state.

A page repository for transferring pages in a computer, such as pagerepository 135 in computer 100, may be a disk drive or other storagemedium, or a virtualized form of these, and may be a component of thecomputer 100. A page repository 135 may be another device or componentcoupled to computer 100, such as a device connected to an IO adapter(e.g., SCSI or Fiber Channel IO adapter) that is a component of (e.g.,plugged into) the computer 100. A page repository 135 may be a networkattached storage (NAS) device or a second computer. An OS may operate asa page repository. For example, OS 102 a may be a virtual IO server andoperate as a page repository 135 to receive pages in use by OS 102 b. AnOS (not shown) operating in another computer (also not shown) coupled tocomputer 100 may act as a page repository 135.

For further understanding the disclosure, “a computer transferringmemory pages” includes components of the computer (e.g., hardwaredevices, an operating system, a hypervisor, or other programs) selectingor transferring the pages. An OS 102 a or 102 b may transfer memorypages to the page repository and may do so at a time or in a manner thata program using a memory page is unaware that the page has beentransferred to the page repository. A hypervisor 120 operating in thecomputer 100 may transfer memory pages to the page repository. Thehypervisor 120 may perform the transfer at a time, or in a manner, thatan OS, or a program, using a memory page is unaware that the page hasbeen transferred to the page repository. Another device (e.g., a secondcomputer) coupled to, or in communication, with computer 100 maytransfer memory pages to a page repository.

The pages of memory associated with an OS may represent a memory imageof the OS and the programs executing within it. A memory image may bethe object of an operation transferring memory pages to a pagerepository. For example, memory image 130 a may contain the memory pagesassociated with OS 102 a, and memory image 130 b may contain the memorypages associated with OS 102 b. An LM operation may transfer the memoryimage 130 b to a page repository 135. A memory image may represent onlypages actually in the memory 130, or may represent a totality of pagesassociated with or in use by an OS or program, including pages notresiding in the memory 130, such as pages paged-out to another pagerepository or paged-out to page repository 135.

FIG. 2 illustrates a computer 200 configured, according to aspects ofthe disclosure, to transfer pages from the computer memory 230 to a pagerepository 235. Computer 200 includes a transfer agent 214, a set ofpages to transfer 212 a and a priority set 212 b of pages to transfer.Computer 200 may transfer pages in these sets to the page repository235. The pages may be pages included in a memory image 130 b; forexample, memory image 130 b may contain pages of an OS 102 b migratingin an LM operation. As another example, pages 250 a and 250 b may bepages in affinity domain 170 to transfer in an AMS operation to makepages 250 a and 250 b available to OS 102 a.

A transfer agent 214 may act to transfer the pages from memory 230 tothe page repository 235. A transfer agent may communicatively couple thememory 230 to the page repository 235 (e.g., directly, by means of anintermediary device, or by means of an IO adapter). A transfer agent 214may be a component of the computer 200 (e.g., hardware or firmware, or acombination thereof), a hypervisor 120, or an OS 102 a and/or 102 b. Atransfer agent 214 may be embodied as a combination of components of thecomputer 200, such as a hypervisor 120 or an OS 102 a and/or 102 binteracting to perform the function of a transfer agent.

A transfer agent 214 may be embodied in a second computer (not shown) ora device (also not shown) coupled to or in communication with thecomputer 200 and the page repository 235, or may in some other mannerhave access to the memory 230 and page repository 235. A transfer agent214 may be wholly or partially embodied in a page repository 235. Forexample, a page repository 235 may be embodied as a VM operating in asecond computer and communicating with computer 200 to access a copy ofthe memory pages to be transferred.

A transfer agent may identify a set of pages to transfer, or may receivea list or description of a set of pages to transfer (e.g., an OS mayprovide a set of pages to a transfer agent), as represented by transferset 212 a including pages one through six. For example, a hypervisor orVM may act as a transfer agent and receive a set of pages to transferfrom an OS. The transfer agent may prepare to transfer the pages in aparticular order, such as first transferring page 1 and thentransferring page 2. The order may be based on metrics associated witheach page, such as LRU or LFU, may be based on criteria such asascending or descending page address, may be based on the inclusion ofpages in particular affinity domains, or may be based on some othercriteria. A variety of criteria to order pages for transfer may beadvantageous to a particular embodiment of a computer 200.

The computer 200 may identify certain pages within the set of pages totransfer, or certain pages within the memory 230 as a whole, that may betransferred prior to other pages. For purposes of understanding thedisclosure, with regard to identifying pages to transfer, ortransferring pages, references to “computer 200” include a particularcomponent (e.g., a hypervisor, firmware, or an OS) of computer 200 or acombination of components of computer 200, including a transfer agent214 or a component thereof.

By way of example, the computer 200 may select pages 250 a, 250 b, 250c, 250 d, 250 e, and 250 f, in memory image 130 b, to transfer to pagerepository 235, and may initially include all of these pages in atransfer set 212 a. For example, these pages may be included in a memoryimage 130 b to migrate in an LM operation and in which page repository235 may be another computer. In another example, these pages may beselected from memory image 130 b to share with another OS, such as 102a, in a memory sharing operation, and page repository 235 may be apaging device or may be a VM. The pages may be ordered for transfer frompage one to page six (page 250 a through 250 f).

The computer 200 may determine that the pages are included in anaffinity domain 170, or are included in an SRAD, such as SRAD 275, thatincludes CPU 210 c. The computer 200 may determine that pages 250 e and250 f have not been accessed by any program, or have been last accessedby a program that is not executing on any CPU within computer 200 (e.g.,the program is in an idle state). Alternatively, the computer 200 maydetermine that pages 250 e and 250 f were last referenced by CPU 210 cand that CPU 210 c is in a suspended state.

Correspondingly the computer 200 may identify pages 250 e and 250 f asless likely to be modified than other pages within the transfer set 212a and may determine to transfer pages 250 e or 250 f, or both, prior totransferring other pages (e.g., 250 a, 250 b, 250 c, or 250 d) intransfer set 212 a.

Similarly, the computer 200 may determine that a page within the set ofpages to transfer, such as page 250 c, was last referenced by program204 b and that program 204 b is executing on a CPU, such as CPU 110 d,that is not included in affinity domain 170, or not included in SRAD275. Correspondingly the computer 200 may identify page 250 c as lesslikely to be modified than pages 250 d, 250 a, or 250 b within thetransfer set 212 a and may determine to transfer page 250 c prior totransferring pages 250 d, 250 a, or 250 b.

In another alternative, computer 200 may determine that page 250 d waslast referenced by program 204 b and that program 204 b is executing ona CPU not included in affinity domain 170, or not included in SRAD 275.Correspondingly the computer 200 may identify page 250 d as less likelyto be modified than pages 250 a or 250 b within the transfer set 212 aand may determine to transfer page 250 d prior to transferring pages 250a or 250 b.

FIG. 3 illustrates a computer 300 configured, according to aspects ofthe disclosure, to transfer pages to a page repository 335 and to a pagerepository 365. The computer 300 includes OSes 102 a, 102 b, and 302 b,and the OSes include programs 104 a in OS 102 a, 104 b in OS 102 b, and304 c in OS 302 b. Computer 300 includes a hypervisor 120 and ahypervisor interface 122. Computer 300 includes CPUs 310 a, 310 b, 310c, and 310 d, and a memory 130, and the memory is organized into pages,such as pages 350 a through 350 d, 340 a through 340 d, and 360 a and360 b. Affinity domain 370 includes the four CPUs 310 a through 310 d,and memory pages one through ten (340 a through 340 d, 350 a through 350d, and 360 a and 360 b).

Pages in the memory may be included in a memory image of a program orOS. Memory image 330 a may be associated with OS 102 a or program 104 aand includes pages 360 a and 360 b. Memory image 330 b may be associatedwith OS 102 b or program 104 b and includes pages 350 a through 350 d.Memory image 330 c may be associated with OS 302 b or program 304 c andincludes pages 340 a through 340 d. Memory images 330 a, 330 b, and 330c are in affinity domain 370.

Page repository 335 may be included in computer 300, or may becommunicatively coupled to computer 300. Similarly, page repository 365may be included in computer 300, or may be communicatively coupled tocomputer 300. In an embodiment, page repository 335 may be, for example,a paging device or a VM to receive pages in a memory sharing operation.Page repository 365 may be a second computer (not shown), or may be acomponent of a second computer, and may receive pages migrated in an LMoperation. A second computer having page repository 365 may be connectedto computer 300 in a variety of ways, including by means of a networkconnection or a directly cabled connection. In another embodiment pagerepository 335 or page repository 365 may be a virtual computer (notshown) included in computer 300, and the virtualized computer may belogically of the form of a computer in the manner of the disclosure. Inembodiments a page repository 335 or 365 may be a physical or virtualdevice of the computer 300.

Computer 300 may include a transfer agent 314 in the manner of thedisclosure. The transfer agent may be communicatively coupled to thepage repository 335, and may be communicatively coupled to the pagerepository 365. The transfer agent 314 may identify memory pages totransfer to both page repository 335 and to page repository 365. Atransfer agent 314 may identify a set of pages to transfer to pagerepository 335, and may include these pages in a transfer set 312 a. Forexample, in an embodiment pages 340 a and 340 b and 350 b and 350 d(pages 1, 2, 6, and 8, respectively, in transfer set 312 a) may be a setof pages in the memory 130 to transfer to page repository 335 in an AMSoperation. Memory image 330 c may be associated with OS 302 b, and OS302 b may be migrating to a second computer (embodying page repository365) in an LM operation. Pages 340 a, 340 b, 340 c, and 340 d(collectively, pages 340), accordingly, may be identified for transferto the second computer.

The transfer agent 314 may determine to include pages 340 in a priorityset 312 b to transfer prior to transferring pages in transfer set 312 a.Transferring pages 340 in priority set 312 b (e.g. as part of an LMoperation) may make the pages in memory 130 corresponding to pages 340available for other purposes, such as sharing in an AMS operation.Correspondingly, as a result of identifying pages in the priority set312 b to transfer, the transfer agent 314 may determine that pages 340a, 340 b, 350 b, and 350 d (pages 1, 2, 6, and 8, respectively, intransfer set 312 a) need not be transferred (e.g., for AMS sharing). Inparticular, pages 340 a and 340 b may not need to be transferred to pagerepository 335 if transferring them first to page repository 365 makesthe corresponding pages in memory 130 available (e.g., for sharing).Similarly, transferring pages 340 c and 340 d to page repository 365 maymake the corresponding pages in memory 130 available (e.g., forsharing), and pages 350 b and 350 d may not need, then, to betransferred to page repository 335. The transfer agent may, in response,remove pages 340 a, 340 b, 350 b, and 350 d from the transfer set 312 a.

It would be evident to one of ordinary skill in the art that computer300 may include a plurality of transfer sets, and a transfer agent mayprocess the transfer sets each individually, or may process the totalityof transfer sets as logically one transfer set. It would be furtherapparent to one of ordinary skill in the art that computer 300 mayinclude a plurality of priority sets and a transfer agent may processthe plurality of priority sets each individually, or may process thetotality of priority sets as logically one priority set.

FIG. 4 illustrates conceptual structures that may be used to recordinformation about CPUs, pages, affinity domains or SRADs, OSes, orprogram threads in the manner of the disclosure. A CPU table 400 mayrecord an affinity domain associated with a set of CPUs included in acomputer. For purposes of understanding the disclosure, an “affinitydomain” may be an affinity domain as defined in the disclosure, or maybe an SRAD. In some embodiments, a CPU table may include an affinitydomain as defined in the disclosure, an SRAD, or both (e.g., includingan SRAD in a column, not shown, of a CPU Table 400).

A CPU table 400 may also record the ID of an OS to which the CPU isallocated, such as CPU 110 a allocated to OS ID 102 a, and CPUs 110 b,110 c, and 110 d allocated to OS ID 102 b. Alternatively, CPU table 400may record a program ID (not shown) to which the CPU is allocated, andmay include an OS ID, a program ID, or both (e.g., including a programID in a column, not shown, of a CPU Table 400). The CPU table 400 mayalso record a state associated with a CPU, and which may include CPUstates such as Active (A) or Suspended (S). For example, CPU table 400records CPUs 110 a, 110 b, and 110 d in an Active state (A), whereas CPU110 c is recorded in a Suspended state (S). In embodiments a transferagent may use a CPU table 400 to determine in which affinity domain (or,SRAD) a CPU is included, to which OS a CPU is allocated, or in whichstate the CPU is operating.

An embodiment may include a page table 402 a or 402 b (collectively,page tables 402) describing elements of a computer associated with amemory page. As illustrated in detail in page table 402 b, a page tablemay include a set of pages (e.g., 150 c through 150 f, 140 a, and 140b), and for each page the page table 402 b may record the identity of anaffinity domain (or, SRAD, or both) in which the page is included. Insome embodiments a page table, such as page table 402 b, may include anaffinity domain as defined in the disclosure, an SRAD, or both (e.g.,including an SRAD in a column, not shown, of a page table 402 b).

A page table 402 b may record the identity of a program that mostrecently accessed a particular page included in the page table. Aprogram, in the context of page table 402 b, may include an OS, aprogram, or a program thread. A page table 402 b may record that theaccess by a program was “local”, from a suspended CPU within theaffinity domain (or, SRAD) that includes the page. Alternatively, a pagetable 402 b may record that the access by a program was “remote”, from aCPU outside of the affinity domain (or, SRAD) of the page. In anembodiment a page table 402 b may be provided for each OS operating in acomputer, such as page table 402 b provided for OS 102 b and page table402 a provided for OS 102 a. Each of page tables 402 a and 402 b mayrecord information with regard to memory pages allocated to thatparticular OS 102 a or 102 b, respectively.

An embodiment may use structures such as or similar to those shown inFIG. 4 to identify pages to transfer to one or more page repositories,in the manner of the disclosure. It would be evident to one of ordinaryskill in the art that the component information of the structuresillustrated in FIG. 4 are of a conceptual nature and may be embodied ina variety of programming or hardware structures suitable for associatingthe components of the tables illustrated, including a single structureor table that may include all the unique elements as they are associatedwith each other.

FIG. 5 exemplifies a method 500 to identify pages to transfer from amemory to one or more page repositories, according to the manner of thedisclosure. For purposes of illustrating the disclosure, and inembodiments, a transfer agent may perform the method. A transfer agentmay include components of a firmware program, a hypervisor, or an OS;or, a transfer agent may include components of any or all of these.Embodiments may perform the method to transfer memory pages to variousembodiments of a page repository, such as other pages in the same orother memory of a computer; another computer, or components thereof; or,another OS operating in the same or another computer. A page repositorymay be some other element or component of or coupled to a computer, suchas an IO adapter, a virtualized IO adapter or IO device, or a networkinterface.

At 502 of method 500 the transfer agent identifies a “transfer set” ofpages in a memory to transfer to a page repository. In embodiments thetransfer set may include pages to page-out to a paging repository, suchas in an AMS or virtual memory operation, or may include pages totransfer to a migration repository, such as in a workload migration oran LM operation. Pages within the transfer set may be allocated to aprogram (e.g., an OS or a program thread), and the program may continueto execute and may access or modify those pages.

At 504 the transfer agent prepares to transfer pages in the transferset. Preparing to transfer pages may include scheduling the pages in thetransfer set for transfer at a later time, or may include beginning toactually transfer the pages in the transfer set. A transfer agent mayorder pages in the transfer set, with respect to each other, fortransfer. For example, the transfer agent may order pages for transferin order of ascending page address, in LRU order, or in LFU order. Atransfer agent may at 504 record the pages comprising the transfer setin a data structure (e.g., a list of transfer pages) and, in the datastructure, may order the pages for transfer with respect to each other.

At 506 the transfer agent identifies a “priority set” of pages withinthe memory to transfer prior to transferring pages in the transfer set.The pages in the priority set may be pages less likely, as compared topages in the transfer set identified at 502, to be modified prior totransferring to a page repository. A transfer agent may, at 506, recordthe priority pages in a data structure, such as a list (e.g., a prioritypage list). In some embodiments, pages in a transfer set and pages in apriority set may be identified for transfer to different pagerepositories, such as pages in the transfer set to transfer to a firstpage repository and pages in the priority set pages to transfer to asecond page repository. For example, pages in the transfer set may bepages to transfer to a paging repository, to make the memory pagesavailable to another program or OS, such as in a virtual memory ormemory sharing (e.g., AMS) operation. Pages identified as priority pagesmay be pages to transfer to a second computer in a memory migrationoperation included in a partition (e.g., LM) or workload migration.

In embodiments, transferring the priority pages may remove the need totransfer some or all of the pages in the transfer set identified at 502.In the foregoing example, transferring the priority (LM) pages beforethe pages in the transfer set (AMS pages), the memory pages occupied bythe priority pages may become available for memory sharing, such thatsome or all of the pages scheduled to page-out for memory sharing (e.g.,pages in the transfer set) need not then be transferred to a pagingrepository.

The priority set may include pages within the transfer set identified at502 to transfer to the same page repository. For example, a transferagent may identify priority pages from among pages in the transfer setare in the same affinity domain as an affinity domain having a CPU in asuspended state, and in which the page was last accessed by a programthat is idle (e.g., not executing). The transfer agent may identifypages to include in the priority set from among pages in the transferset that were last accessed by a program, in which the program isexecuting on a CPU in an affinity domain different from that whichincludes the page.

A transfer agent may, at 506, order pages for transfer within a priorityset. For example, the priority pages may have relative priorities withrespect to each other, such as some priority pages having “priority one”(first to transfer), and other priority pages having “priority two”(e.g., pages that may be transferred subsequent to priority one pages).A transfer agent may identify a page last referenced by an idle programas a priority one page, and may identify a page last referenced by aprogram executing on a CPU in a different affinity domain than that pageto be a priority two page. In other embodiments a transfer agent mayselect a particular page from pages within a priority page listaccording to other criteria such as LRU or LFU relative to other pagesin the priority set.

At 508 the transfer agent selects a priority page to transfer to a pagerepository. At 510 the transfer agent determines if the page selected at508 has been already transferred. For example, the priority pageselected at 508 may be included in a transfer set and, as a result of504, the transfer agent, or a component of the computer, may havealready transferred the priority page. If at 510 the transfer agentdetermines that the selected priority page has not already beentransferred, the transfer agent at 514 prepares the transfer of theselected priority page to occur prior to transferring pages prepared fortransfer at 504. Preparing to transfer the priority page may includescheduling the transfer of the page for a later time, or may includeactually transferring the page.

Alternatively, if at 510 the transfer agent determines that the selectedpriority page has already been transferred, at 512 the transfer agentdetermines whether or not the selected priority page (from 508) has beenmodified since the time it had been transferred. If the priority pagehas been modified then at 514 the transfer agent prepares to transferthe modified priority page again. If at 512 the transfer agentdetermines that the priority page has not been modified, the transferagent at 514 prepares the priority page for transfer. If the prioritypage is included in the transfer set, the transfer agent may, at 514,remove the priority page from the transfer set.

At 516 the transfer agent determines if there are other pages remainingin the priority set to transfer. If so, the transfer agent returns to508, selects another priority page from the priority set, and at 510 to514 processes the selected priority page. If at 516 the transfer agentdetermines that there are no more priority pages remaining in thepriority set, the transfer agent at 518 completes processing thetransfer of pages in the priority or transfer set. Processing thetransfer of the pages at 518 may include actually transferring thepages, or may include scheduling the pages for transfer at a later time.

While the disclosure of FIG. 5 illustrates a transfer agent performingthe method 500, the method is not limited to, nor is the disclosureintended to limit the method to, performance by a transfer agent asdefined in the disclosure. It would be apparent to one of ordinary skillin the art that a variety of elements or components of a computer, inthe manner of the disclosure, individually or in combination, mayperform the method 500.

FIG. 6 illustrates an example method 600, according to features of thedisclosure, to identify a set of affinity domains having CPUs in asuspended state. Affinity domains, in the example method 600, may beaffinity domains as defined in the disclosure, or may be SRADs. Themethod of FIG. 6 may be suitable in an embodiment to identify prioritypages to transfer to page repositories in the manner of the disclosure.For purposes of illustration, and in embodiments of the disclosure, atransfer agent may perform the method 600.

At 602 of method 600 the transfer agent identifies a transfer set ofmemory pages to transfer to a page repository and determines that pagesin the transfer set are allocated to an OS. At 604 the transfer agentdetermines a “set A” comprising CPUs also allocated to the OS. At 604,the transfer agent may alternatively determine that the CPUs areallocated to a particular program or program thread of the OS, or may beallocated to any particular program executing in the computer, includinga program executing in firmware or a hypervisor. For purposes ofillustrating method 600 according to the features of the disclosure,“OS” is representative of any of the foregoing types of programs.

At 606 the transfer agent identifies a “set B” that has as members CPUsfrom set A that are in a suspended (e.g., folded) state. At 608 thetransfer agent further identifies a “set C” that has as members CPUsallocated to the OS that are not suspended. Set C may be computed as theset difference of set A (CPUs allocated to the OS) minus set B (CPUsallocated to the OS that are in a suspended state). At 610 the transferagent may identify a “set D” that has as members affinity domains thatinclude CPUs in set C. Continuing at 612, the transfer agent identifiesa set E that has as members affinity domains of the set A CPUs (all CPUsallocated to the OS). At 614 the transfer agent identifies a set F thathas as members affinity domains of the suspended CPUs in set B. The setF may be computed as the set difference of set E minus set D.

While the disclosure of FIG. 6 illustrates a transfer agent performingthe method 600, the method is not limited to, nor is the disclosureintended to limit the method to, performance by a transfer agent asdefined in the disclosure. It would be apparent to one of ordinary skillin the art that a variety of elements or components of a computer, inthe manner of the disclosure, individually or in combination, mayperform the method 600.

FIG. 7 illustrates an example method 700 to identify a set of prioritypages to transfer to a page repository in the manner of the disclosure.For purposes of illustration, and in embodiments of the disclosure, atransfer agent may perform of the method 700.

At 702 a transfer agent identifies a first set of memory pages totransfer to a page repository. A transfer agent may identify pages toinclude in the first set in the manner disclosed in reference to 502 ofFIG. 5. At 704 the transfer agent identifies a set of suspended CPUsand, at 706, the transfer agent identifies affinity domains containingthe suspended CPUs. A transfer agent may use the method 600, or aspectsthereof, to determine suspended CPUs and affinity domains includingthem. Affinity domains, in the example method 700, may be affinitydomains as defined in the disclosure, or may be SRADs.

At 708 through 724 the transfer agent determines if one or more pages inthe first set may be priority pages in the manner of the disclosure. At708 the transfer agent selects a page from the first set of memorypages. At 710 the transfer agent determines whether or not the selectedpage is in an affinity domain of a suspended CPU, such as an affinitydomain identified at 706. If the selected page is not in an affinitydomain having a suspended CPU, the transfer agent, at 710, determinesthat the selected page is not a priority page and proceeds, at 726, todetermine if there are additional pages to transfer. If, on the otherhand, the selected page is in an affinity domain having a suspended CPU,the transfer agent, at 712, determines the identity of a program thatlast accessed the selected page. At 714 the transfer agent determineswhether the identified program is idle or not. At 712 the transfer agentmay, alternatively, determine that no program has accessed the selectedpage and, accordingly, at 722, may include the selected page in a set of“priority one” pages.

If the transfer agent determines, at 714, that the program from 712 isnow in an idle state (e.g., not executing, or dispatched to execute, onany CPU), the transfer agent, at 722, includes the selected page in aset of “priority one” pages. If the program from 712 is not idle, at 716the transfer agent further determines if the program identified at 712is executing (or, may be dispatched to execute and execution is yetpending) on a CPU in the affinity domain that includes the page selectedat 708. If the program is executing (or, dispatched to execute) on aprocessor in the same affinity domain as the selected page, the transferagent, at 716, determines that the selected page is not a priority pageand at 726 determines if there are more pages to transfer.

If, instead, at 716, the transfer agent determines that the program from712 is executing (or, may be dispatched to execute and execution is yetpending) on a CPU in an affinity domain other than that which includesthe page from 708, at 718 the transfer agent determines if the programlast accessed the selected page using a CPU in the same affinity domainas the selected page (i.e., the access was “local”). If the access waslocal the transfer agent, at 724, includes the selected page in a set of“priority two” pages. If, on the other hand, the access was not local,at 720 the transfer agent includes the selected page in a set of“priority three” pages.

At 726 the transfer agent determines if there are pages remaining in thetransfer set to process according to the method 700. If so, the transferagent returns to 708 to select a new page from among those remaining andprocess that page according to 710 through 724. If the transfer agentdetermines, at 726, that there are no pages remaining to process, at 728the transfer agent prepares transfer of priority pages to precedetransfer of other pages in the first set of memory pages from 702.

In preparing pages for transfer to a page repository a transfer agentmay determine that transfer of priority pages should precede transfer ofother pages in a set of pages to transfer, and may prepare transfer ofthe priority pages accordingly. A transfer agent may prepare transfer ofpriority one pages to precede transfer of priority two pages, and mayprepare transfer of priority two pages to precede transfer of prioritythree pages. The method of FIG. 7 may be suitable to an embodiment inthe manner of the disclosure, to identify priority pages to transfer inan LM operation, to transfer in an AMS operation, or to transfer in acombination of such operations occurring concurrently or sequentiallywithin a computer.

It would be evident to one of ordinary skill in the art as long aprogram continues to operate (e.g., has not terminated, whether or notthe program is idle), a page selected at 708 and transferred as a resultof 728 (or transferred as a result of 518 of method 500 in FIG. 5), maybe later modified by the program. A transfer agent may, accordingly,need to detect such modification and may need to repeat the transfer ofthat page.

FIG. 8 illustrates an example method 800 to identify sets of pages totransfer to a plurality of page repositories, in the manner of thedisclosure. For purposes of illustration, and in embodiments of thedisclosure, a transfer agent may perform of the method 800.

At 802 the transfer agent identifies a “transfer set one” of pages in amemory to transfer to a first page repository. Various embodiments mayidentify the pages to transfer to the first repository for a variety ofpurposes, such as disclosed previously. In some embodiments the pagesmay be identified as pages to transfer to make the memory page availablein an AMS or virtual memory operation. In other embodiments the pagesmay be identified as pages to transfer to checkpoint or suspend an OS orprogram. The pages included in the transfer set one may be allocated toa program that may modify those pages.

At 804 the transfer agent prepares the transfer of pages in the transferset one to the first page repository. Preparing the transfer of thepages may include scheduling transfer of a page for a later time, or mayinclude initiating the actual transfer of a page. Preparing to transfera page may include inserting the page into a list of pages that acomponent of a computer may operate on to transfer pages.

At 806 a transfer agent identifies a “transfer set two” of pages in amemory to transfer to a second page repository. Various embodiments mayidentify the pages to transfer to the second repository for a variety ofpurposes, such as disclosed previously. In some embodiments the pagesmay be identified as pages to transfer in an LM operation. In otherembodiments the pages may be identified as pages to transfer tocheckpoint or suspend an OS or program. Pages included in the transferset two may be included also in transfer set one. Pages included in thetransfer set two may be allocated to a program that may modify thosepages.

At 808 the transfer agent determines that pages in transfer set two maybe transferred to the second page repository prior to transferring pagesin transfer set one to the first page repository. Accordingly, at 808the transfer agent selects a page from transfer set two. At 810 thetransfer agent determines whether or not the selected page has alreadybeen transferred to the second page repository (e.g., as having beenalready transferred prior to commencing 808). If not, at 814 thetransfer agent prepares the page for transfer.

If, at 810, the transfer agent determines, instead, that the selectedpage has already been transferred to the second page repository, at 812the transfer agent determines whether or not the selected page has beenmodified since the time it had been transferred. If the selected pagehas not been modified, at 816 the transfer agent removes the selectedpage from transfer set two. If the selected page is included also in thetransfer set one, at 816 the transfer agent may also remove the selectedpage from transfer set one. For example, the selected page may beincluded in transfer set two to transfer to the second repository in anLM operation, and may also be included in transfer set one to transferto the first page repository as part of an AMS operation.

If, at 812, the transfer agent determines that the selected page hasbeen modified, at 814 the transfer agent prepares transfer of theselected page to the second page repository. At 814 the transfer agentmay prepare transfer of the pages in transfer set two to occur in aparticular order. For example, the transfer agent may order the transferof the pages according to a relative memory page address, a LRU metric,or an LFU metric. A transfer agent may order the transfer of theselected page according to the methods of the disclosure. If theselected page is included also in the transfer set one, at 816 thetransfer agent may also remove the selected page from transfer set one.

At 818, the transfer agent determines if there are more pages remainingin transfer set two to process according to 810 through 816. If so, thetransfer agent returns to 808 and at 808 selects another page from thoseincluded in transfer set two and processes the selected page accordingto 810 through 816. If, at 818, the transfer agent determines that thereare no more pages remaining in transfer set two to process, at 820 thetransfer agent processes pages prepared for transfer at 804 and 814.Processing, at 820, may include actually transferring page in transferset two and then actually transferring pages remaining in transfer setone. Processing, at 820, may also include scheduling the transfer ofpages in transfer set two to occur prior to transfer of pages intransfer set one. A transfer agent may order the transfer of theselected page according to the methods of the disclosure.

Also at 820, a transfer agent may determine that pages in transfer settwo that are prepared for transfer, or actually being transferred, tothe second page repository may remove the need to transfer one or morepages included in transfer set one. Accordingly, the transfer agent maydetermine to not transfer one or more pages in transfer set one to thefirst page repository and may remove those pages from transfer set one.For example, pages included in transfer set one may be pages to transferin an AMS operation. Pages in transfer set two may include, for example,pages to transfer in an LM operation. Transferring pages in transfer settwo may make alternative pages available in the memory for the AMSoperation and, correspondingly, a transfer agent may remove pages intransfer set one that are pages to transfer in an AMS operation.

The structures and methods of the disclosure are illustrative of themanner of embodiments of the disclosure involving a plurality of pagerepositories. However, these are not intended to limit embodiments toonly two page repositories. Rather, it would be apparent to one ofordinary skill in the art that other embodiments in the manner of thedisclosure may incorporate more than two page repositories, and themanner in which to accordingly modify the embodiments according toaspects of the disclosure.

FIG. 9 depicts an article of manufacture or computer program product 900that is an embodiment of the disclosure. The computer program product900 may include a recording medium 902, and the recording medium 902 maystore program modules 904, 906, 908, and 910 for a computer to carry outthe aspects of the disclosure. The recording medium 902 may be a CD ROM,DVD, tape, diskette, non-volatile or flash memory, storage mediumaccessed by a network connection, or other similar computer readablemedium for containing a program product.

A sequence of program instructions within, or an assembly of one or moreinterrelated modules defined by, the program modules 904, 906, 908, or910 may direct a computer to implement the aspects of the disclosureincluding, but not limited to, the structures and methods illustrated inthe disclosure.

The disclosure may be a system, a method, and/or a computer programproduct at any possible technical detail level of integration. Thecomputer program product may include a computer readable storage medium(or media) having computer readable program instructions thereon forcausing a processor to carry out aspects of the disclosure.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present disclosure may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the disclosure.

Aspects of the disclosure are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause the computer, other programmable apparatus, or other device toperform a series of operational steps to produce a computer implementedprocess, such that the instructions which execute on the computer, otherprogrammable apparatus, or other device implement the functions/actsspecified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The descriptions of the various embodiments of the present disclosurehave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

What is claimed is:
 1. A method for transferring pages from a memory,the method comprising: identifying a first affinity domain, the firstaffinity domain associating a first set of processors with a first setof affinity pages in the memory, none of the processors included in thefirst set of processors in a suspended state; identifying a first memorypage, the first memory page included in the first set of affinity pages,the first memory page to be transferred to a page repository;identifying a second affinity domain, the second affinity domainassociating a second set of processors with a second set of affinitypages in the memory, the second set of processors including a firstprocessor, the first processor in the suspended state; identifying asecond memory page, the second memory page included in the second set ofaffinity pages, the second memory page to be transferred to the pagerepository, the second memory page associated with a first program;determining that the first program is not executing; determining totransfer the second memory page before transferring the first memorypage, the determining based on the first memory page in the firstaffinity domain, the second memory page in the second affinity domain,the second memory page associated with the first program, and the firstprogram not executing; and preparing transfer of the second memory pageto occur prior to transfer of the first memory page.
 2. The method ofclaim 1 further comprising: identifying a third memory page, the thirdmemory page included in the second affinity domain, the third memorypage associated with a second program, the third memory page to betransferred to the page repository; determining that the second programwas the last program to access the third memory page; determining thatthe second program last accessed the third page using a secondprocessor, the second processor included in the second affinity domain,the second processor in the suspended state; determining that the secondprogram is not executing on any processor in the second affinity domain;determining to transfer the third memory page before transferring thefirst memory page, the determining based on the first memory page in thefirst affinity domain, the third memory page in the second affinitydomain, the third memory page last accessed by the second program usingthe second processor, and the second program not executing on anyprocessor in the second affinity domain; and preparing transfer of thethird memory page to occur prior to transfer of the first memory page.3. The method of claim 1 further comprising: identifying a third memorypage, the third memory page included in the second affinity domain, thethird memory page associated with a second program, the third memorypage to be transferred to the page repository; determining that thesecond program was the last program to access the third memory page;determining that the second program is not executing on any processor inthe second affinity domain; determining that the second program lastaccessed the third page using a second processor, the second processornot included in the second affinity domain; determining to transfer thethird memory page before transferring the first memory page, thedetermining based on the first memory page in the first affinity domain,the third memory page in the second affinity domain, the second programnot executing on any processor in the second affinity domain, and thethird memory page last accessed by the second program using the secondprocessor; and preparing transfer of the third memory page to occurprior to transfer of the first memory page.
 4. The method of claim 1wherein the method is performed by at least one of a firmware program,an operating system, a hypervisor, DMA hardware, and a virtualizeddevice.
 5. The method of claim 1 wherein the page repository includes atleast one of the memory, a storage medium, an IO adapter, a deviceincluded in or coupled to an IO adapter, a virtualized IO adapter ordevice, a network attached storage device, an operating system, and asecond computer.
 6. A computer program product for transferring pagesfrom a memory, the computer program product comprising a computerreadable storage medium having program instructions embodied therewith,the program instructions executable by a first computer to perform amethod comprising: identifying, by the first computer, a first affinitydomain, the first affinity domain associating a first set of processorswith a first set of affinity pages in the memory, none of the processorsincluded in the first set of processors in a suspended state;identifying, by the first computer, a first memory page, the firstmemory page included in the first set of affinity pages, the firstmemory page to be transferred to a page repository; identifying, by thefirst computer, a second affinity domain, the second affinity domainassociating a second set of processors with a second set of affinitypages in the memory, the second set of processors including a firstprocessor, the first processor in the suspended state; identifying, bythe first computer, a second memory page, the second memory pageincluded in the second set of affinity pages, the second memory page tobe transferred to the page repository, the second memory page associatedwith a first program; determining, by the first computer, that the firstprogram is not executing; determining, by the first computer, totransfer the second memory page before transferring the first memorypage, the determining based on the first memory page in the firstaffinity domain, the second memory page in the second affinity domain,the second memory page associated with the first program, and the firstprogram not executing; and preparing, by the first computer, transfer ofthe second memory page to occur prior to transfer of the first memorypage.
 7. The computer program product of claim 6, the method furthercomprising: identifying, by the first computer, a third memory page, thethird memory page included in the second affinity domain, the thirdmemory page associated with a second program, the third memory page tobe transferred to the page repository; determining, by the firstcomputer, that the second program was the last program to access thethird memory page; determining, by the first computer, that the secondprogram last accessed the third page using a second processor, thesecond processor included in the second affinity domain, the secondprocessor in the suspended state; determining, by the first computer,that the second program is not executing on any processor in the secondaffinity domain; determining, by the first computer, to transfer thethird memory page before transferring the first memory page, thedetermining based on the first memory page in the first affinity domain,the third memory page in the second affinity domain, the third memorypage last accessed by the second program using the second processor, thesecond program not executing on any processor in the second affinitydomain; and preparing, by the first computer, transfer of the thirdmemory page to occur prior to transfer of the first memory page.
 8. Thecomputer program product of claim 6, the method further comprising:identifying, by the first computer, a third memory page, the thirdmemory page included in the second affinity domain, the third memorypage associated with a second program, the third memory page to betransferred to the page repository; determining, by the first computer,that the second program was the last program to access the third memorypage; determining, by the first computer, that the second program is notexecuting on any processor in the second affinity domain; determining,by the first computer, that the second program last accessed the thirdpage using a second processor, the second processor not included in thesecond affinity domain; determining, by the first computer, to transferthe third memory page before transferring the first memory page based onthe first memory page in the first affinity domain, the third memorypage in the second affinity domain, the second program not executing onany processor in the second affinity domain, and the third memory pagelast accessed by the second program using the second processor; andpreparing, by the first computer, transfer of the third memory page tooccur prior to transfer of the first memory page.
 9. The computerprogram product of claim 6 wherein the method is performed, in the firstcomputer, by at least one of a firmware program, an operating system, ahypervisor, DMA hardware, and a virtualized device.
 10. The computerprogram product of claim 6 wherein the page repository includes at leastone of the memory, a storage medium, an IO adapter, a device included inor coupled to an IO adapter, a virtualized IO adapter or device, anetwork attached storage device, an operating system, and a secondcomputer.
 11. A system for transferring pages from a memory, the systemcomprising: the memory; and processors in communication with the memory,wherein the system is configured to perform a method, the methodcomprising: identifying a first affinity domain, the first affinitydomain associating a first set of processors with a first set ofaffinity pages in the memory, none of the processors included in thefirst set of processors in a suspended state; identifying a first memorypage, the first memory page included in the first set of affinity pages,the first memory page to be transferred to a page repository;identifying a second affinity domain, the second affinity domainassociating a second set of processors with a second set of affinitypages in the memory, the second set of processors including a firstprocessor, the first processor in the suspended state; identifying asecond memory page, the second memory page included in the second set ofaffinity pages, the second memory page to be transferred to the pagerepository, the second memory page associated with a first program;determining that the first program is not executing; determining totransfer the second memory page before transferring the first memorypage, the determining based on the first memory page in the firstaffinity domain, the second memory page in the second affinity domain,the second memory page associated with the first program, and the firstprogram not executing; and preparing transfer of the second memory pageto occur prior to transfer of the first memory page.
 12. The system ofclaim 11, the method further comprising: identifying a third memorypage, the third memory page included in the second affinity domain, thethird memory page associated with a second program, the third memorypage to be transferred to the page repository; determining that thesecond program was the last program to access the third memory page;determining that the second program last accessed the third page using asecond processor, the second processor included in the second affinitydomain, the second processor in the suspended state; determining thatthe second program is not executing on any processor in the secondaffinity domain; determining to transfer the third memory page beforetransferring the first memory page, the determining based on the firstmemory page in the first affinity domain, the third memory page in thesecond affinity domain, the third memory page last accessed by thesecond program using the second processor, and the second program notexecuting on any processor in the second affinity domain; and preparingtransfer of the third memory page to occur prior to transfer of thefirst memory page.
 13. The system of claim 11, the method furthercomprising: identifying a third memory page, the third memory pageincluded in the second affinity domain, the third memory page associatedwith a second program, the third memory page to be transferred to thepage repository; determining that the second program was the lastprogram to access the third memory page; determining that the secondprogram is not executing on any processor in the second affinity domain;determining that the second program last accessed the third page using asecond processor, the second processor not included in the secondaffinity domain; determining to transfer the third memory page beforetransferring the first memory page, the determining based on the firstmemory page in the first affinity domain, the third memory page in thesecond affinity domain, the second program not executing on anyprocessor in the second affinity domain, and the third memory page lastaccessed by the second program using the second processor; and preparingtransfer of the third memory page to occur prior to transfer of thefirst memory page.
 14. The system of claim 11 wherein the method isperformed by at least one of a firmware program, an operating system, ahypervisor, DMA hardware, and a virtualized device.
 15. The system ofclaim 11 wherein the page repository includes at least one of thememory, a storage medium, an IO adapter, a device included in or coupledto an IO adapter, a virtualized IO adapter or device, a network attachedstorage device, an operating system, and a second computer.